Skip empty files in G3Reader #137
Conversation
Avoid throwing an IO error when an empty G3 file is included in the input filenames by silently moving on to the next file in the list until reaching a non-empty file or the end of the list. Fixes #134.
arahlin
force-pushed
the
skip_empty_files
branch
from
27a73a7 to
684baa8
Compare
|
Per the comment on the issue, this was a deliberate choice as a way to fail loudly on failed cluster jobs. Can we hide the new behavior behind an option so that people who turn it on are aware of consequences? |
Sure, that's probably a good idea. @samtomguns are you ok with this solution? |
I'll add that the current failure mode is inconsistent -- if the only input file is empty, or just the first file in the list is empty, then the code runs without failure, without this fix. |
|
is something special about the last file from the perspective of cluster jobs? wouldn't it make more sense to have it fail on every empty file unless some extra option is turned off? |
|
I think the right solution then is to check whether a file emitted any frames, and raise an error if it didn't (or raise a warning and skip to the next file, if the user sets that option) |
|
This already fails on the presence of any empty file. We can change the behavior, of course -- I just wanted everyone to be on the same page about why the current behavior exists before we do. |
|
Sorry, that was wrong. That was what it should have done. Since it isn't, let's just merge the PR. |
|
Sounds good. As a compromise, I've added an error message for when the reader encounters an empty file. |
Avoid throwing an IO error when an empty G3 file is included in the input filenames by moving on to the next file in the list, until reaching a non-empty file or the end of the list. Issues an error message for every empty file that is encountered.
Fixes #134.